Papers with classification metrics
ToVo: Toxicity Taxonomy via Voting (2025.findings-naacl)
Copied to clipboard
Tinh Son Luong, Thanh-Thien Le, Thang Viet Doan, Linh Ngo Van, Thien Huu Nguyen, Nguyen Thi Ngoc Diep
| Challenge: | Existing toxic content detection models face limitations due to the closed-source nature of training data and the paucity of explanations for their evaluation mechanism. |
| Approach: | They propose a mechanism that integrates voting and chain-of-thought processes to produce a high-quality open-source dataset for toxic content detection. |
| Outcome: | The proposed model improves transparency and customizability while facilitating better fine-tuning for specific use cases. |
FACT-AUDIT: An Adaptive Multi-Agent Framework for Dynamic Fact-Checking Evaluation of Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Existing fact-checking evaluation methods rely on static datasets and classification metrics, which fail to evaluate justification production and uncover the nuanced limitations of LLMs. |
| Approach: | They propose a framework that adaptively and dynamically assesses LLMs’ fact-checking capabilities by incorporating justification production alongside verdict prediction. |
| Outcome: | Experiments show that the framework differentiates among state-of-the-art LLMs, providing valuable insights into model strengths and limitations in model-centric fact-checking analysis. |
PR-XAI: PageRank-Based Feature Attribution for Transformers (2026.acl-long)
Copied to clipboard
| Challenge: | Existing feature attribution methods for transformer models suffer from limitations that undermine their efficacy. |
| Approach: | They propose a feature attribution method for transformer models based on PageRank . they propose attribution methods that apply PageRank to attention-derived graphs . |
| Outcome: | The proposed method outperforms state-of-the-art methods in faithfulness and classification metrics with significant gains on long-form text. |
Validating Automatic Evaluation of Controllable Counterspeech Generation: Rankings Matter More Than Scores (2026.eacl-long)
Copied to clipboard
| Challenge: | Existing methods for evaluating attributes of counterspeech are limited and the validity of such evaluations is questionable when the classifiers themselves have only modest performance. |
| Approach: | They examine the automatic evaluation of counterspeech attributes using a multi-attribute counterseech dataset containing 2,728 samples. |
| Outcome: | The proposed model can be trusted by classifier validation, and it can rank models with confidence. |
The Effects of Unimodal Representation Choices on Multimodal Learning (L18-1)
Copied to clipboard
| Challenge: | In the real world, multiple modes of information are gathered to create knowledge in a way humans can understand. |
| Approach: | They propose to combine unimodal representations to map multiple modes of information to a single space . they argue that the way they are combined can affect performance and classification metrics . |
| Outcome: | The proposed model can be used to correlate words in a textual description of an object with multimodal representations. |
Classification-based Quality Estimation: Small and Efficient Models for Real-world Applications (2021.emnlp-main)
Copied to clipboard
| Challenge: | Sentence-level Quality estimation (QE) is traditionally a regression task . but large multilingual contextualized language models are expensive and infeasible for real-world applications. |
| Approach: | They evaluate several model compression techniques for QE and find they are inefficient . they argue that a full model parameterization is required to achieve SoTA results . |
| Outcome: | The proposed models are poorly expressive in a regression task, the authors argue . they show that reframing QE as a classification problem and evaluating models would improve their performance in real-world applications. |
Sentence Embedding Leaks More Information than You Expect: Generative Embedding Inversion Attack to Recover the Whole Sentence (2023.findings-acl)
Copied to clipboard
| Challenge: | Sentence-level representations are beneficial for various natural language processing tasks. |
| Approach: | They propose a generative embedding inversion attack that reconstructs input sequences based only on their sentence embeddeds. |
| Outcome: | The proposed model outperforms previous embedding inversion attacks in classification metrics and generates coherent and contextually similar sentences as original inputs. |
Not What the Doctor Ordered: Surveying LLM-based De-identification and Quantifying Clinical Information Loss (2025.emnlp-main)
Copied to clipboard
Kiana Aghakasiri, Noopur Zambare, JoAnn Thai, Carrie Ye, Mayur Mehta, J Ross Mitchell, Mohamed Abdalla
| Challenge: | De-identification is an application of NLP where automated algorithms remove identifying information of patients and providers. |
| Approach: | They propose to use generative large language models to de-identify patients and providers . they propose to validate existing metrics to quantify extent of inappropriate removal . |
| Outcome: | The proposed method is based on a survey of LLM-based de-identification research . it shows that the models perform poorly in identifying clinically relevant changes . |